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ABSTRACT 

The  recently  released  DoD  Unmanned  Aerial  Vehicles 
Roadmap  [P]  discusses  advancements  in  UAV  autonomy  in 
terms  of  autonomous  control  levels  (ACL).  The  ACL  concept 
was  pioneered  by  researchers  in  the  Air  Force  Research 
Laboratory’s  Air  Vehicles  Directorate  who  are  charged  with 
developing  autonomous  air  vehicles.  In  the  process  of 
developing  intelligent  autonomous  agents  for  UAV  control 
systems  we  were  constantly  challenged  to  ''tell  us  how 
autonomous  a  UAV  is,  and  how  do  you  think  it  can  be 
measured.. U  Usually  we  hand- waved  away  the  argument  and 
hoped  the  questioner  will  go  away  since  this  is  a  very  subjective, 
and  complicated,  subject,  but  within  the  last  year  we’ve  been 
directed  to  develop  national  intelligent  autonomous  UAV  control 
metrics  -  an  IQ  test  for  the  flyborgs,  if  you  will.  The  ACL  chart 
is  the  result.  We’ve  done  this  via  intense  discussions  with  other 
government  labs  and  industry,  and  this  paper  covers  the  agreed 
metrics  (an  extension  of  the  OODA  -  observe,  orient,  decide,  and 
act  -  loop)  as  well  as  the  precursors,  “dead-ends”,  and  out-and- 
out  flops  investigated  to  get  there. 

Keywords:  autonomy  metrics,  machine  intelligence 
metrics,  UAV,  autonomous  control 

1.  Background 

At  top  levels  of  the  US  Department  of  Defense  an  effort 
has  been  initiated  to  coordinate  researchers  across  the 
Services  and  industry  in  meeting  national  goals  in  fixed- 
wing  vehicle  development.  The  Fixed-Wing  Vehicle 
Initiative  (FWV)  has  broad  goals  across  numerous  vehicle 
technologies.  One  of  those  areas  is  mission  management 
of  UAVs.  Our  broad  goal  is  to  develop  the  technology 
allowing  UAVs  to  replace  human  piloted  aircraft  for  any 
conceivable  mission.  This  implies  that  we  have  to  give 
UAVs  some  level  of  autonomy  to  accomplish  the 
missions.  One  of  the  cornerstones  of  the  FWV  process  is 
the  establishment  of  metrics  so  one  know  that  a  goal  is 
reached,  but  what  metrics  were  available  for  measuring 
UAV  autonomy?  Our  research,  in  conjunction  with 
industry,  determined  that  there  was  not  any  sort  of  metric 
as  we  desired.  Thus  we  set  out  to  define  our  own  [Note 
1]. 

But  what  characteristics  should  these  metrics  have?  We 
decided  that  they  needed  to  be: 

•  Easily  visualized  such  that  upper  management  could 
grasp  the  concepts  in  a  couple  of  briefing  slides. 

•  Broad  enough  to  measure  past,  present  and  future 
autonomous  system  development. 

•  Have  enough  resolution  to  easily  track  impact  of 
technological  program  investments. 


So,  they  had  to  be  simple,  apply  to  a  broad  range  of 
systems,  and  yet  exhibit  good  resolution.  Obviously  a 
simple  task,  but  first  let’s  look  at  what  it  means  to  be 
autonomous. 

2.  Quick  Difference  Between  Autonomous 
and  Automatic  (our  definition) 

Many  people  don’t  realize  that  there  is  a  significant 
difference  between  the  words  autonomous  and  automatic. 
Many  news  and  trade  articles  use  these  words 
interchangeably.  Automatic  means  that  a  system  will  do 
exactly  as  programmed,  it  has  no  choice.  Autonomous 
means  that  a  system  has  a  choice  to  make  free  of  outside 
influence,  i.e.,  an  autonomous  system  has  free  will.  For 
instance,  let’s  compare  functions  of  an  automatic  system 
(autopilot)  and  an  autonomous  guidance  system: 

•  Autopilot:  Stay  on  course  chosen. 

•  Autonomous  Guidance:  Decide  which  course  to  take, 
then  stay  on  it. 

Example:  a  cruise  missile  is  not  autonomous,  but 
automatic  since  all  choices  have  been  made  prior  to 
launch. 

3.  We  Need  To  Measure  Autonomy,  Not 
Intelligence 

For  some  reason  people  tend  to  equate  autonomy  to 
intelligence.  Looking  through  the  proceedings  of  the  last 
NIST  Intelligent  Systems  Workshop  there  are  several 
papers  which  do  this,  and  in  fact,  the  entire  conference 
sets  the  tone  that  “intelligence  is  autonomy”  [3].  They  are 
not  the  same.  Many  stupid  things  are  quite  autonomous 
(bacteria)  and  many  very  smart  things  are  not  (my  3  year 
old  daughter  seemingly  most  of  the  time).  Intelligence 
(one  of  a  myriad  of  definitions)  is  the  capability  of 
discovering  knowledge  and  using  it  to  do  something. 
Autonomy  is: 

•  the  ability  to  generate  one’s  own  purposes  without 
any  instruction  from  outside  (L.  Fogel) 

•  having  free  will  (B.  Clough) 

What  we  want  to  know  is  how  well  a  UAV  will  do  a  task, 
or  better  yet,  develop  tasks  to  reach  goals,  when  we’re  not 
around  to  do  it  for  the  UAV.  We  really  don’t  care  how 
intelligent  it  is,  just  that  it  does  the  job  assigned. 
Therefore,  intelligence  measures  tell  us  little.  So,  although 
we  could  talk  about  the  Turing  Test  [7]  and  other 
intelligence  metrics,  that  is  not  what  we  wanted. 
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4.  Well,  It  Should  Be  Easy  To  Find  Metrics, 
One  Has  The  Web  And  Other  Info  Sources, 
Right? 

Well,  one  would  think  so,  but  after  an  exhaustive  one- 
month  search  involving  the  author.  Air  Force  Research 
Laboratory  Library  Staff,  and  several  other  search 
organizations  we  found  two.  Two.  Now  if  the  goal  was 
“find  machine  intelligence  metrics”  we  would  have  been 
inundated  with  piles  of  paper.  In  addition  to  the 
aforementioned  workshop,  we  would  be  looking  through 
hundreds  of  publications  and  papers.  Maybe  it  was  a  good 
thing  we  were  looking  for  autonomy  metrics! 

We  had  thought  that  maybe,  just  maybe,  the  folks  working 
distributed  autonomous  robotic  systems  had  looked  at  this 
problem,  but  our  questions  to  experts  in  that  field  revealed 
that  they  are  just  starting  to  ask  those  questions  themselves 
[4].  The  problem  the  researchers  have  in  this  area  is  that 
the  metrics  they  are  coming  up  with  are  task  specific  - 
they  don’t  have  general  metrics  quantitatively  measuring 
higher-level  characteristics  of  autonomous  robot  control 
architectures. 

So  what  were  the  two  that  we  found?  Los  Alamos 
National  Laboratory’s  “Mobility,  Acquisition,  and 
Protection”  space  [6],  and  Draper  Laboratory’s  “Three 
Dimensional  Intelligence  Space”[7].  The  following  is  a 
short  discussion  of  each  [Note  5]. 

1.  Los  Alamos  National  Lab:  Mobility,  Acquisition, 
and  Protection  (MAP) 

MAP  comes  from  the  lab  of  Mark  Tilden,  who  develops 
simple  robots  based  on  analog  circuits  [7^].  He  needed  a 
way  to  quantify  the  autonomous  nature  of  his  systems,  and 
teamed  up  with  LANL  Physicist  Brosl  Hasslacher  to 
develop  the  “Mobility,  Acquisition,  and  Protection”  space, 
or  MAP  for  short.  Figure  1  is  a  diagram  of  MAP  from  [d] 

As  one  might  expect  from  the  name,  this  method  uses 
mobility,  acquisition,  and  protection  to  measure  the  ability 
of  an  autonomous  system  to  survive  in  the  world. 

•  Mobility  relates  to  the  capability  of  utilizing 
movement  in  the  environment.  MO  implies  no  motion 
abilities  where  as  M3  can  move  in  three  dimensions, 
and  M-  means  that  external  force  must  be  used  to 
move  object. 

•  Acquisition  relates  to  the  ability  to  gather,  store,  and 
utilize  energy.  AO  implies  zero  energy  consumption 
or  delivery,  A4  means  planned  tactics  used  to 
efficiently  extract,  store,  and  utilize  external  energy, 
while  A-  indicates  object  uses  a  non-replenishable 
energy  store 

•  Protection  indicates  the  capability  to  defend  oneself 
P-  indicates  one  is  physically  more  fragile  than  the 
environment  while  PI  means  one  executes  flight/hide 
behaviors  against  hostile  stimuli,  and  P3  demonstrates 
tactical  fight/flight  behaviors. 


Figure  1:  MAP  Survival  Space  in  which  autonomous 
systems  can  be  measured 


These  level  metrics  are  fully  described  in  Table  1  below. 
The  space  has  three  metrics  has  the  three  metrics  outlines 
above,  and  six  levels. 


Leve  1 

M  o  b  ility 

M  etric 

Acquis  itio  n 

P  rotection 

M  otion  0  n  ly  0  ecu  rs 
Under  Application 
Of  An  External 

F  0  rce 

Operates  from  a  non- 
replentishable  energy 
source  (battery,  power 
line ,  etc.) 

N  egative 

D  efe  n  s  ive 
capabilities 
(physically  m  ore 
fragile  than  the 
environm  ent) 

0 

No  M  0 tio n  Ab ilities 

Zero  energy 

CO n s u  m  p tio n  or 
de  live  ry 

Zero  defensive 
abilities  (structu ral 
strength  equal  to 
environm  ent 

1 

Moves  Deliberately 
In  One  D  im  ension 

Can  directly 
extract/apply  external 
energy  when  available 

Flight  or  hide 
behavior  against 
hostile  stim  ulus 

2 

Moves  D  e  lib  e  rate  ly 
in  T wo  D  im  ensions 

Can  effciently 
extra  ct/s  to  re /utilize 
external  energy 

F  ig h t  or  flig h t 
behavior  against 
hostile  stim  ulus 

3 

Moves  D  e  libe  rate  ly 
In  T  h  ree 

D  im  ensions 

Uses  focused  tactics  to 
efficiently  extract,  store 
and  utilize  external 
energy 

T  actica  I  fig  h t/flig  h  t 
behavior  against 
hostile  stim  ulus 

4 

Capable  Of  dual¬ 
modem  otion  with 
tools,  vehicles,  or 
a p p licatio n  of 
specific  design 
e le m  e nts 

Uses  planned  tactics 
to  efficiently  extract, 
store  and  utilize 
external  energy 

T 00,  vehicle,  or 
m  ate ria I  use  in 
fight/flight  tactics 

5 

Human 

Human 

Human 

Table  1:  Level  Descriptors  For  MAP  Survival  Space 


MAP  is  actually  quite  a  versatile  visual  tool,  allowing 
disparate  items  to  be  plotted  on  the  same  page.  Since  there 
are  three  metrics,  one  can  use  a  “radar  chart”  to  display 
the  measurements  of  a  particular  autonomous  system,  and 
this  is  excellent,  since  upper  management  likes  radar 
charts!  The  Los  Alamos  researchers  also  realized  this  and 
included  a  MAP  radar  chart  in  their  report.  Showing  this 
versatility,  one  can  plot  an  ant,  human,  and  a  toaster  on 
the  same  chart  as  is  done  in  Figure  2!  Tilden  and 
Hasslacher  successfully  use  MAP  to  illustrate  the  survival 
capabilities  of  the  robots  they  design. 


Figure  2:  Various  Objects  plotted  in  MAP  survival 
Space . 

Can  this  be  used  to  measure  the  autonomy  of  UAVs? 
Possibly.  Figure  3  shows  a  plot  of  a  multi-UAV  neural 
net-based  autonomous  control  system.  This  plot  illustrates 
the  limitations  of  MAP  for  our  use.  All  UAVs  would 
score  M3  and  A-  -  they  move  in  three  dimensions  but 
require  stored  fuel.  Protection  ranges  between  PO 
(structural  strength  to  absorb  damage)  to  P4  (groups  of 
UAVs  deliberately  take  out  SAM  sites).  So,  is  this  really 
useful  for  FWV  autonomy  measurement?  No. 

•  Only  one  axis  shows  any  variability,  the  others  are 
fixed. 

•  The  metrics  just  do  not  address  operational 
characteristics  of  UAVs.  They  do  not  relate  the 
autonomy  present  in  the  vehicle  to  the  capability  to 
perform  useful  missions. 

•  The  metrics  do  not  address  interaction  between  UAVs 
(teams,  swarms,  etc.) 

•  The  metrics  do  not  allow  us  to  adequately 
discriminate  between  different  levels  of  autonomy. 
For  instance,  an  RPV  and  an  UAV  with  autonomy 
doing  the  same  mission  would  score  the  same. 

So  although  using  MAP  seems  to  make  sense  for  simple 
robots,  as  a  UAV  autonomy  measurement  it  isn’t 
particularly  useful. 

So,  the  first  metric  space  we  examined  could  not  fulfill  our 
autonomous  control  system  metric  search,  so  we  went  on 
to  investigate  the  other  candidate  we  found  -  the  “3D 
Intelligence  Space”  of  Charles  Stark  Draper  Laboratory. 

2.  Draper  3D  Intelligence  Space 

Charles  Stark  Draper  Laboratory  (Cambridge,  MA)  has 
been  developing  robotic  systems  for  military  and  other 
Federal  customers  for  a  number  of  years.  They  saw  the 
same  need  to  measure  how  well  their  systems  could 
perform  various  tasks,  and  developed  metrics  under  the 
sponsorship  of  the  Office  of  Naval  Research. 
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Figure  3:  Autonomous  Control  System  Plotted  On 
MAP 

These  metrics  were  described  in  a  paper  [7]  written  by  the 
Draper  Lab  researchers  last  year.  This  paper  contained 
several  different  options  to  measure  both  intelligence  and 
autonomy.  Here  we  focus  on  the  3D  Intelligence  Space 
outlined  in  Table  2. 

One  can  see  this  metric  space  has  a  couple  desireable 
attributes: 

•  It  has  three  metrics,so  we  can  still  use  three-axis  radar 
charts  to  represent  the  results,  which  will  keep 
management  happy(Figure  CSDl). 

•  It  has  metrics  which  can  be  directly  related  to 
operational  issues. 


M  etric 


Level 

M  obility  Control 

T ask  Planning 

Situational  Awareness 

1 

None,  RPA  Only 

None,  RPA  Only 

None,  RPA  only,  or 
sensor  as  conduit 

Low-level  sensor 

2 

0  perator  Assisted 

W  aypoint  or 
featu re  oriented 

processing  ,  e.g.  visual 
servoing  (tern  p late 

tracking ) 

3 

Get  to  waypoint,  do 
one  feature-based 
com  m  and 

Interpret  goals 
into  action 

Single-Sensor  model 
m  atch  ing 

4 

Integ rate  m  u Itip le 
actions 

M  u  Iti-Ag  ent 
Collaboration  and 
02 

Integrated,  multi-sensor 
fusion 

Table  2:  3D  Intelligence  Space 

Note  that  we  made  the  distance  between  levels  in  Figure  4 
increasing  exponentially  to  represent  the  difficulties 
technicallly  in  going  between  steps. 

We  went  ahead  an  plotted  the  same  multi-UAV 
autonomous  control  systems  used  earlier  in  evaluation 
MAP  space  on  the  Draper  radar  chart.  The  results  of  this 
are  in  Figure  5.  Note  that  this  simple  multi-UAV 
autonomous  control  system  managed  to  “max-out”  the 
metric  space  on  all  three  axes,  and  highlight  the  fact  that 
the  resolution  needs  to  be  better.  Other  drawbacks 
include: 

•  Task  Planning  axis  needs  to  be  renamed.  Many 
successful  autonomous  systems  are  based  on  pure 
reactive  behaviors  (such  as  insects).  Task  planning 
isn’t  a  prerequisite  to  autonomy,  it  just  allows  better 
reactions  to  complicated  situations. 


•  Situational  awareness  is  based  on  the  number  of 
sensors  and  how  they  are  fused,  not  on  whether  or  not 
the  autonomous  system  understands  what’s  going  on 
around  it.  In  other  words,  this  should  be  a 
measurement  on  how  well  the  “big  picture”  is 
comprehended  and  understood. 

Chart 

Situation 
Awareness 


Task  Pianning 

Figure  4:  Radar  Chart  Of  Draper  Metrics 

The  drawbacks  not  withstanding,  the  Draper  metrics 
provided  us  another  good  way  of  looking  at  the  world. 


Multi-UAV Autonomous  Neural  Net  SW 


Task  Pianning 

Figure  5:  Autonomous  Control  System  Plotted  Using 
Draper  Metrics 

5.  Initial  Autonomous  Control  Level  (ACL) 
Chart 

We  decided  that  since  no  existing  metric  space  existed  that 
could  be  directly  used,  we  would  integrate  the  best 
features  of  the  ones  we  found  with  what  we  already  used 
internally  to  represent  where  our  technology  was  going. 
Table  3  is  that  first  cut  at  an  Autonomous  Control  Level 
(ACL)  chart  [Note  2]. 


Level 

Level  Descriptor 

Percep  tIon/SItua  tional  A  wareness 

Analysis/Decislon  Making 

Communication/Cooperation 

10 

Human-LIke 

9 

Multi-Vehicle 

Tactical 

Performance 

Optimization 

Detection  &  tracking  of  other  air  vehicles  within 
airspace 

Full  decision  making  capability  on-board 

Dynamically  optimize  multi-ship  group  for  tactical 
situation 

Distributed  cooperation  with  other  air  vehicles 

On-board  deconfliction  and  collision  avoidance 

Fully  independent  of  supervision/control  if 
desired;  No  centralized  control  within  multi-UAV  group 

8 

Multi-Vehicle 

Mission 

Performance 

Optimization 

Detection  &  tracking  of  other  air  vehicles  within 
local  airspace 

OK  to  operate  in  controlled  airspace  w/o  external 
control 

Continuous  mission/trajectory  evaluation  &  replan  - 
optimize  for  current  mission  situation 

Avoid  collisions  and  replan/optimize  trajectory  to  meet 
goals,  etc 

External  supervision  -  abort/recall  or  new  overall  goal 
On-board  deconfliction  &  collision  avoidance 

Distributed  cooperation  with  other  A/V's 

7 

Real-Time 

Multi-Vehicle 

Cooperation 

Detection  of  other  A/V's  in  local  airspace 

Multi-threat  detection/analysis  on-board 

Continuous  flight  path  evaluation  &  replan 

Compensate  for  anticipated  system  malfunctions, 
weather,  etc  -  optimize  trajectory  to  meet  goals, 
manage  resources,  avoid  threats,  etc 

On-board  collision  avoidance 

Uses  off-board  data  sources  for  deconfliction  &  tracking 
Hierarchical  cooperation  with  other  A/V's 

6 

Real-Time 

Multi-Vehicle 

Coordination 

Detection  of  other  A/V's  in  local  airspace 

Single  threat  detection/analysis  on-board 

Event-driven  on-board,  RT  flight  path  replan  -  goal 
driven  &  avoid  threats 

RT  Health  Diagnosis;  Ability  to  compensate  for  most 
failures  and  flight  conditions  -  inner  loop  changes 
reflected  in  outer  loop  performance 

On-board  collision  avoidance 

Uses  off-board  data  sources  for  deconfliction  &  tracking 
Assumed  acceptance  of  replan;  External  supervision  - 
rejection  of  plan  is  exception 

Possible  close  air  space  separation  (1-100  yds) 

5 

Fault/Event 

Adaptive 

Vehicle 

Automated  Aerial  Refueling  &  Formation  sensing 
Situational  awareness  supplemented  by  off-board  data 
(threats,  other  A/Vs,  etc) 

Event-driven  on-board,  RT  traj  replan  to  new  destinatior 
RT  Health  Diagnosis;  Ability  to  compensate  for  most 
failures  and  flight  conditions;  Ability  to  predict  onset  of 
failures  (e.g.  Prognostic  Health  Mgmt) 

On-board  assessment  of  status  vs  trajectory 

On-board  derived  vehicle  trajectory  "corridors" 

Uses  off-board  data  sources  for  deconfliction  &  tracking 
External  supervision  -  accept/reject  of  replan 

Possible  close  air  space  separation  (1-100  yds)  for 

AAR,  formation  in  non-threat  conditions 

4 

Robust  Response 
to 

Anticipated 

Faults/Events 

Threat  sensing  on-board 

RT  Health  Diagnosis  (Can  1  continue  with  these 
problems?);  Ability  to  compensate  for  most 
failures  and  flight  conditions  (e.g.  Adaptive  inner  loop 
control);  Automatic  trajectory  execution;  On-board 
assessment  of  status  vs  mission  completion 

Secure,  within  LOS  electronic  tether  to  nearby  friendlies 
Offboard  derived  vehicle  "corridors";  Medium  vehicle 
airspace  separation  (100's  of  yds) 

Threat  analysis  off-board 

3 

Limited  Response 
to  Real  Time 
Faults/Events 

RT  Health  Diag  (What  is  the  extent  of  the  problems?) 
Ability  to  compensate  for  limited  failures  (e.g. 
Reconfigurable  Control) 

Automatic  trajectory  execution 

Health  Status  monitored  by  external  supervision 
Off-board  replan;  Waypoint  plan  upload 

Wide  airspace  separation  requirements  (miles) 

2 

Pre-loaded 
Alternative  Plans 

RT  Health  diagnosis  (Do  1  have  problems?) 

Automatic  trajectory  execution  (via  waypoints) 

Preloaded  alternative  plans  (e.g.  abort) 

External  commands  -  alternative  plans,  approvals, 
aborts 

Reports  status  on  request  or  on  schedule 

Wide  airspace  separation  requirements  (miles) 

1 

Execute 

Preplanned 

Mission 

Situational  awareness  via  Remote  Operator 

Flight  Control  and  Navigation  Sensing 

Robotic/Preprogrammed 

Pre/Post  Flight  BIT 

External  control  via  low  level  commands 

Reports  status  on  request 

Wide  airspace  separation  requirements  (miles) 

No  on-board  knowledge  of  other  air  vehicles  -  all 
actions  are  preplanned 

0 

Remotely 

Piloted 

Vehicle 

Flight  Control  (attitude,  rates)  sensing 

Nose  camera 

Situational  awareness  via  Remote  Pilot 

N/A 

Remotely  Piloted 

Vehicle  status  data  via  telemetry 

Table  3:  Initial  ACL  Metrics  Chart 


We  kept  three  metrics  since  we  liked  the  idea  of 
representing  systems  as  areas  on  a  radar  chart  when 
briefing  management.  We  added  ten  levels  for  better 
resolution  between  remotely  piloted  aircraft  and  fully 
autonomous  UAVs.  The  metrics  related  to  operational 
issues  while  still  being  attached  somewhat  to 
technological  systems.  Populating  the  levels  was  a  group 
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Figure  6:  Autonomous  Control  System  Plotted  On  Initial 
ACL  Radar  Chart 


endeavor,  with  a  team  of  researchers,  program  managers, 
tech  area  leads,  and  contractor  experts  determining  the 
meaning  of  each  level:  “1”  was  simple  -  the  traditional 
remotely  piloted  aircraft  (RC-type)  while  “10”  was 
“human”.  The  trick  was  populating  the  eight  levels 
between.  As  with  the  Draper  metric  space  in  Figure  4  we 
represented  the  radar  chart  as  having  levels  which  are 
exponentially  more  difficult.  We  then  plotted  the  same 
multi-UAV  autonomous  control  system  as  before,  and  the 
result  is  in  Figure  6.  We  recovered  some  of  the  resolution 
lost  in  the  Draper  metrics;  however,  we  still  had  some 
issues  with  the  metrics: 


•  The  metrics  weren’t  broad  enough  to  cover  UAVs 
acting  on  strategic  knowledge  to  achieve  strategic 
results.  Our  chart  limited  them  to  tactical  thinking 
only. 

•  One  metric  mixed  cooperation  with  communication  - 
mixing  a  “what”  with  a  “how”. 

In  general,  we  tried  to  stuff  as  much  into  three  metrics  as 
possible  to  retain  simplicity  in  briefing  presentation  and  in 
the  process  lost  the  capability  to  split  out  issues  of  multi- 
UAV  control  and  human-UAV  interaction,  to  name  two. 

We  were  going  to  press  ahead  and  use  this  metric  space 
when  one  of  the  autonomous  control  system  development 
engineers  came  up  with  a  good  idea  [Note  3]. 


6.  If  You’re  Replacing  A  Human,  Why  Not 
Measure  Like  One? 

The  great  insight  was  this:  we  are  designing  algorithms, 
agents  if  you  will,  to  replace  pilot  decision  functions. 
Machines  replace  human  -  so  why  not  look  to  the  human 


effectiveness  community  for  metrics?  Modify  the  OODA 
(observe,  orient,  decide  &  act)  loop  -  originally  developed 
to  show  how  to  get  inside  your  enemy’s  decision  loops  - 
[8,  &  Note  4]  for  our  use,  and  populate  the  levels  with 
modifications  of  the  qualifiers  of  the  initial  ACL  chart. 
Table  4  is  what  we  developed  using  this  insight.  The 
same  team  of  experts  that  developed  the  initial  ACL  chart 
also  worked  on  the  new  ACL  chart  to  ensure  consistency 
with  earlier  thoughts.  We  lost  the  three  axes 
representation,  which  means  we  lost  the  ability  to  generate 
the  “simple”  radar  charts  which  makes  management 
happy;  however,  we  gained  better  resolution  between 
metrics  which,  at  least  in  our  thoughts,  more  than  made  up 
for  that. 

Since  we  have  developed  the  ACL  chart,  we’ve  used  it  to 
both  assess  the  current  UAV  efforts,  and  to  extract  from 
that  where  our  own  technical  efforts  must  go.  Nationally, 
we’ve  developed  time-phased  autonomous  system  goals  to 
put  our  autonomous  systems  roadmap  together.  The  ACL 
has  been  published  as  part  of  the  DoD  UAV  Roadmap  [9], 
and  other  DoD  Labs  use  it  to  measure  their  autonomy 
development.  Locally  we’ve  developed  technical  area 
roadmaps  putting  programs  together  to  meet  the  time- 
phased  ACL  goals.  The  ACL  chart  also  acts  as  a  program 
advocacy  tool,  allowing  us  to  show  management  how  each 
program  fits  into  increasing  ACL  capability  for  each 
metric,  and  also  how  each  program  investment  integrates 
into  the  national  strategy.  Our  experience  from  using  it 
for  one  budget  planning  cycle  has  been  very  positive: 

•  Once  management  was  briefed  on  the  chart  and  it’s 
development  (and  some  in  management  had 
ownership  in  it’s  development)  it  was  accepted  as  the 
tool  to  measure  program  goals. 

•  It  provided  clear  indications  of  where  the  technology 
was  targeted  and  what  national  goal  it  met,  allowing 
better  informed  budget  planning  decisions. 

•  We  have  common  ground  for  talking  amongst  other 
Federal  technology  development  organizations, 
universities,  and  industry.  Each  of  us  has  a  much 
clearer  picture  as  to  where  technological  programs  fit. 

7.  Summary 

Our  work  with  autonomous  UAVs  indicated  to  us  that  we 
needed  metrics  to  measure  the  progress  of  our  programs 
building  that  autonomy.  The  same  issues  existed  on  a 
national  level,  so  we  decided  to  develop  metrics  for  the 
national-level  effort,  then  apply  those  to  our  local  program 
planning  process. 

Our  literature  search  for  autonomous  system  metrics  only 
returned  two  references  for  metrics.  Both  we  examined 
and  used  on  an  example  problem.  Although  each  wasn’t 
directly  applicable,  concepts  of  each,  integrated  with  our 
own  existing  ideas,  formed  an  initial  ACL  chart.  This 
chart  was  modified  based  on  concepts  human  dynamists 
had  developed  -  specifically  the  OODA  loop.  The 


resulting  set  of  metrics  captured  our  original  intent.  [Note 

4] 

Since  development,  the  ACL  metrics  have  been  used 
successfully  at  the  Air  Force  Research  Laboratory  in 
developing  plans  and  programs  in  autonomous  UAV 
controls  research.  The  ACL  chart  is  in  current  review  at 
DOD  levels  to  be  applied  across  the  services  as  part  of  the 
FVW  initiative.  With  this  development  we  are  pressing 
ahead  in  the  assessment  of  possible  sub-metrics  to  better 
hone  our  program  planning. 
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9.  Notes 

1.  Not  that  we  didn’t  have  any.  We  had  already  split 
autonomy  into  four  levels  depending  on  the  amount  of 
human  interaction  and  where  it  occurred.  These  are: 

•  Remotely  Piloted:  The  UAV  is  simply  a  remotely 
piloted  aircraft  with  the  human  operator  making  all 
decisions. 

•  Remotely  Operated:  The  human  allows  the  UAV  to 
do  the  piloting,  but  outer  loop  decisions  are  made  by 
the  human  (like  where  to  go  and  what  to  do  once 
there).  The  UAV  is  a  “mother-may-I”  system,  asking 
the  human  permission  to  do  tasks. 

•  Remotely  Supervised:  The  human  allows  the  UAV  to 
execute  its  own  tasks,  only  taking  command  if  the 
UAV  fails  to  properly  execute  them. 

•  Fully  Autonomous:  The  UAV  receives  goals  from  the 
humans  and  translates  that  into  tasks  which  it  does 
without  human  intervention.  The  UAV  has  authority 
to  make  all  decisions. 

2.  Most  of  the  grunt  work  in  putting  the  chart  together 
was  done  by  Dan  Thompson,  AFRL/VACC,  and  Dr. 
Alan  Burkhard,  AFRL/VAC  -  the  rest  of  us  got  to 
snipe  at  it. 

3.  The  researcher’s  name  is  Bob  Smith,  and  besides 
coming  up  with  decent  ideas  he  also  has  developed  a 
formation  flight  agent  for  UAV  formations  which 
uses  a  blend  of  deliberate  and  emergent  behavior. 

4.  Boyd’s  OODA  loop,  originally  developed  to  illustrate 
how  to  take  advantage  of  an  enemy,  has  been  grasped 
wholeheartedly  by  business  management  folks.  The 
observe,  assess,  design,  and  act  (OADA)  loop 
organizational  dynamists  use  to  explain  how  decisions 
are  made  is  a  direct  descendant. 

5.  I  know,  you’re  wondering  about  Sheridan’s 
Autonomy  Levels.  Truth  of  the  matter  is  that  if  you 
search  for  “autonomy”,  “metrics”,  “measuring 
autonomy”,  etc.,  you  don’t  run  into  Sheridan.  Had  I 
searched  using  “teleoperation”  I  would  have  found 
the  Sheridan  Autonomy  Levels  [77]. 


Perception/Situational  Awareness 


Orient 

Analysis/Coordination 


Decide 

Decision  Making 


Act 

Capability 


Fuiiy  Autonomous  Cognizant  of  all  within  Battlespace 


Coordinates  as  necessary 


Capable  of  total  indepenance 


Requires  little  guidance  to  do  job 


Battlespace  inference  -  Intent  of  self  and  others 
Battiespace  (allies  and  foes). 

Swarm  Complex/Intense  environment  -  on-board  tracking 

Cognizance 


Strategic  group  goals  assigned 
Enemy  strategy  inferred 


Distributed  tactical  group  planning 
Individual  determination  of  tactical  goal 
Individual  task  planning/execution 
Choose  tactical  targets 


Group  accomplishment  of  strategic  goal  with 
no  supervisory  assistance 


Proximity  inference  -  Intent  of  self  and  others 
(allies  and  foes) 


Strategic  group  goals  assigned 


Coordinated  tactical  group  planning 
Individual  task  planning/execution 


Group  accomplishment  of  strategic  goal  with 
minimal  supervisory  assistance 


Cognizance 

Reduced  dependence  upon  off-board  data 

Enemy  tactics  inferred 

Choose  targets  of  opportunity 

(example:  go  SCUD  hunting) 

ATR 

7 

Short  track  awareness  -  History  and  predicti\e  battlesp 

Tactical  group  goals  assigned 

Individual  task  planning/execution  to  meet  goals 

Group  accomplishment  of  tactical  goal  with 

Battiespace 

data  in  limited  range,  timeframe,  and  numbers 

Enemy  trajectory  estimated 

minimal  supervisory  assistance 

Knowiedge 

Limited  inference  supplemented  by  off-board  data 

6 

Reai  Time 

Tactical  group  goals  assigned 

Coordinated  trajectory  planning  and  execution  to  meet 

Group  accomplishment  of  tactical  goal  with 

Muiti-Vehicie 

Ranged  awareness  -  on-board  sensing  for  long  range. 

Enemy  location  sensed/estimated 

goals  -  group  optimization 

minimal  supervisory  assistance 

Cooperation 

supplemented  by  off-board  data 

Possible  close  airspace  separation  (1-100  yds) 

5 

Reai  Time 

Sensed  awareness  -  Local  sensors  to  detect  others. 

Tactical  group  plan  assigned 

On-board  trajectory  replanning  -  optimizes  for 

Group  accomplishment  of  tactical  plan  as  externally 

Muiti-Vehicie 

Fused  with  off-board  data 

RT  Health  Diagnosis;  Ability  to  compensate  for  most 

current  and  predicti\e  conditions 

assigned 

Coordination 

failures  and  flight  conditions;  Ability  to  predict  onset  of 

Collision  awidance 

Air  collision  awidance 

failures  (e.g.  Prognostic  Health  Mgmt) 

Possible  close  air  space  separation  (1-100  yds)  for 

Group  diagnosis  and  resource  management 

AAR,  formation  in  non-threat  conditions 

4 

Deliberate  awareness  -  allies  communicate  data 

Tactical  plan  assigned 

On-board  trajectory  replanning  -  e\ent  dri\en 

Self  accomplishment  of  tactical  plan  as  externally 

Fauit/Event 

Assigned  Rules  of  Engagement 

Self  resource  management 

assigned 

Adaptive 

RT  Health  Diagnosis;  Ability  to  compensate  for  most 

Deconfliction 

Vehicie 

failures  and  flight  conditions  -  inner  loop  changes 

reflected  in  outer  loop  performance 

Medium  \ehicle  airspace  separation  (100's  of  yds) 

3 

Robust  Response 

Health/status  history  &  models 

Tactical  plan  assigned 

E\aluate  status  \s  required  mission  capabilities 

Self  accomplishment  of  tactical  plan  as  externally 

to  Reai  Time 

RT  Health  Diag  (What  is  the  extent  of  the  problems?) 

Abort/RTB  if  insufficient 

assigned 

Fauits/Events 

Ability  to  compensate  for  most  control  failures  and 

flight  conditions  (i.e.  adapti\e  inner-loop  control) 

I 

2 

Health/status  sensors 

RT  Health  diagnosis  (Do  I  ha\e  problems?) 

Execute  preprogrammed  or  uploaded  plans 

Self  accomplishment  of  tactical  plan  as  externally 

Changeabie 

Off-board  replan  (as  required) 

in  response  to  mission  and  health  conditions 

assigned 

Mission 

1 

Execute 

Preloaded  mission  data 

Prepianned 

Flight  Control  and  Navigation  Sensing 

Pre/Post  Flight  BIT 

Preprogrammed  mission  and  abort  plans 

Wide  airspace  separation  requirements  (miles) 

Mission 

Report  status 

0 

Remoteiy 

Flight  Control  (attitude,  rates)  sensing 

Telemetered  data 

N/A 

Control  by  remote  pilot 

Piioted 

Nose  camera 

Remote  pilot  commands 

Vehicie 

Table  4:  Final  ACL  Chart 


